Image processing device and electronic device

ABSTRACT

An image processing device includes: an input unit, first image data being a portion of an image with first and second photographic subjects being imaged being inputted therein, the second subject to be displayed after the first, and the first subject again to be displayed by repeating control to shift a portion of the image displayed upon the display unit in a first direction and to display a portion not displayed; and an image generation unit generating, from the first image data, second image data including the first and second subjects, and the second subject is arranged towards the first direction from the first subject, if a first distance by which the image displayed shifts from the first subject being displayed until the second subject is displayed is longer than a second distance by which the image displayed shifts from the second subject until the first subject is displayed.

TECHNICAL FIELD

The present invention relates to an image processing device and to anelectronic device.

BACKGROUND ART

A camera that performs cutting out, display and recording of a portionof a photographic image photographed by an ultra wide angle camera isper se known (for example, refer to PTL1).

CITATION LIST Patent Literature

-   PTL1: Japanese Laid-Open Patent Publication No. 2012-119804.

SUMMARY OF INVENTION

According to a first aspect, an image processing device, comprises: aninput unit through which are inputted first image data which is aportion of an image in which a first photographic subject and a secondphotographic subject are imaged, and which is employed for the secondphotographic subject to be displayed after the first photographicsubject has been displayed, and for the first photographic subject thenagain to be displayed upon the display unit by repeating control toshift a portion of the image displayed upon the display unit in a firstdirection and to display a portion of the image that is not displayedupon the display unit; and an image generation unit that generates, fromthe first image data, second image data including the first photographicsubject and the second photographic subject, and in which the secondphotographic subject is arranged towards the first direction from thefirst photographic subject, if a first distance by which the imagedisplayed upon the display unit shifts from the first photographicsubject being displayed upon the display unit until the secondphotographic subject is displayed upon the display unit is longer than asecond distance by which the image displayed upon the display unitshifts from the second photographic subject being displayed upon thedisplay unit until the first photographic subject is displayed upon thedisplay unit.

According to a second aspect, an image processing device, comprises: aninput unit through which are inputted first image data which is aportion of an image in which a first photographic subject and a secondphotographic subject are imaged, and which is employed for the secondphotographic subject to be displayed after the first photographicsubject has been displayed, and for the first photographic subject thenagain to be displayed upon the display unit by repeating control toshift a portion of the image displayed upon the display unit in a firstdirection and to display a portion of the image that is not displayedupon the display unit; and an image generation unit that generates, fromthe first image data, second image data including the first photographicsubject and the second photographic subject, based on a first distanceby which the image displayed upon the display unit shifts from the firstphotographic subject being displayed upon the display unit until thesecond photographic subject is displayed upon the display unit, and asecond distance by which the image displayed upon the display unitshifts from the second photographic subject being displayed upon thedisplay unit until the first photographic subject is displayed upon thedisplay unit

According to a third aspect, an electronic device, comprises: a displayunit that displays an image in which a first photographic subject and asecond photographic subject are imaged; a control unit that displays thesecond photographic subject after the first photographic subject hasbeen displayed, and then again displays the first photographic subjectupon the display unit, by repeating control to shift a portion of theimage displayed upon the display unit in a first direction and todisplay a portion of the image that is not displayed upon the displayunit; and an image generation unit that generates image data includingthe first photographic subject and the second photographic subject, andin which the second photographic subject is arranged towards the firstdirection from the first photographic subject, if a first distance bywhich the image displayed upon the display unit shifts from the firstphotographic subject being displayed upon the display unit until thesecond photographic subject is displayed upon the display unit is longerthan a second distance by which the image displayed upon the displayunit shifts from the second photographic subject being displayed uponthe display unit until the first photographic subject is displayed uponthe display unit.

According to a fourth aspect, an electronic device, comprises: a displayunit that displays first image data in which a first photographicsubject and a second photographic subject are imaged; a control unitthat displays the second photographic subject after the firstphotographic subject has been displayed, and then displays the firstphotographic subject again upon the display unit, by repeating controlto shift a portion of the first image data displayed upon the displayunit in a first direction and to displays a portion of the first imagedata that is not displayed upon the display unit; and an imagegeneration unit that generates, from the first image data, second imagedata in which the first photographic subject and the second photographicsubject are arranged based on a first distance by which the imagedisplayed upon the display unit shifts from the first photographicsubject being displayed upon the display unit until the secondphotographic subject is displayed upon the display unit, and a seconddistance by which the image displayed upon the display unit shifts fromthe second photographic subject being displayed upon the display unituntil the first photographic subject is displayed upon the display unit.

According to a fifth aspect, an image processing device, comprises: aninput unit through which is inputted an all-around image including afirst photographic subject and a second photographic subject that havebeen imaged by an imaging unit; and an image generation unit that takesa direction from the first photographic subject towards the secondphotographic subject as being a first direction in a partial image ofthe all-around image in which the first photographic subject, the secondphotographic subject, and a third photographic subject that is presentin a shortest path from the first photographic subject to the secondphotographic subject are included, and generates from the allaround-image an image that includes the first photographic subject andthe second photographic subject, and in which the second photographicsubject is arranged towards the first direction from the firstphotographic subject.

According to a sixth aspect, an image processing device, comprises: aninput unit through which is inputted an all-around image including afirst photographic subject and a second photographic subject that havebeen imaged by an imaging unit; and an image generation unit thatgenerates from the all-around image an image in which the firstphotographic subject and the second photographic subject are arrangedbased on a shortest path in the all-around image from the firstphotographic subject to the second photographic subject.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram schematically showing the structure of animage processing system;

FIG. 2 is a block diagram schematically showing the structure of animaging device;

FIG. 3 shows schematic illustrations showing an image capturing range ofan imaging unit and an omnidirectional image;

FIG. 4 shows block diagrams schematically showing the structure of animage processing device and the structure of a reproduction device;

FIG. 5 shows figures for explanation of processing for reproduction ofan omnidirectional image;

FIG. 6 shows figures for explanation of processing for creation of a twodimensional image;

FIG. 7 shows figures showing examples of a two dimensional image;

FIG. 8 is a flow chart for processing for creation of a two dimensionalvideo;

FIG. 9 shows schematic figures showing a variant embodiment of anomnidirectional image;

FIG. 10 is a block diagram schematically showing an electronic devicethat serves both as an image processing device and a reproductiondevice;

FIG. 11 is a figure for explanation of processing for creation of a twodimensional image;

FIG. 12 shows figures for explanation of processing for creation of atwo dimensional image; and

FIG. 13 shows figures for explanation of processing for creation of atwo dimensional image.

DESCRIPTION OF EMBODIMENTS First Embodiment

FIG. 1 is a block diagram schematically showing the structure of animage processing system 1. This image processing system 1 comprises animaging device 2, an image processing device 3, and a reproductiondevice 4. The imaging device 2 is an electronic device such as, forexample, a digital camera, a smart phone, a tablet terminal, or thelike. The image processing device 3 is an electronic device such as, forexample, a digital camera, a smart phone, a tablet terminal, a personalcomputer, or the like. And the reproduction device 4 is an electronicdevice such as, for example, a digital camera, a smart phone, a tabletterminal, a personal computer, a digital photo frame, a head mounteddisplay, or the like.

The imaging device 2 has a still image imaging function and a videoimaging function. The still image imaging function is a function forcapturing an omnidirectional or entire sphere image (as will bedescribed hereinafter). And the video imaging function is a function forrepeatedly capturing omnidirectional images and creating anomnidirectional video, each frame of which is one of theseomnidirectional images. From this omnidirectional video that has beencreated by the imaging device 2, the image processing device 3 creates atwo dimensional video (to be described hereinafter), each frame of whichis a two dimensional image whose angle of view is narrower than that ofthe corresponding omnidirectional image. And the reproduction device 4replays (or displays) omnidirectional images or two dimensional videos.

Explanation of the Imaging Device 2

FIG. 2 is a block diagram schematically showing the structure of theimaging device 2. The imaging device 2 comprises an imaging unit 20, afirst image capturing optical system 21, a second image capturingoptical system 22, and a storage unit 23. And the imaging unit 20comprises a first imaging element 201 and a second imaging element 202.

The first image capturing optical system 21 and the second imagecapturing optical system 22 are so-called fisheye lenses. The firstimage capturing optical system 21 forms an image upon the imagingsurface of the first imaging element 201 of a photographic subject overa hemispherical range. To put it in another manner, the first imagingelement 201 is adapted to be capable of capturing an image over a rangeof 360° in the horizontal direction and over a range of 180° in thevertical direction. This image capturing range of the first imagingelement 201 is termed the “first hemisphere”.

And the second image capturing optical system 22 forms an image upon theimaging surface of the second imaging element 202 of a photographicsubject over a hemispherical range that is different from the firsthemispherical range. To put it in another manner, the second imagingelement 202 is adapted to be capable of capturing an image over a rangeof 360° in the horizontal direction and 180° in the vertical direction.This image capturing range of the second imaging element 202 is termedthe “second hemisphere”.

Together, the first hemisphere and the second hemisphere make up acomplete sphere. In other words, by employing the first imaging element201 and the second imaging element 202, the imaging unit 20 forms animage over the range of a complete sphere, 360° in the horizontaldirection and 360° in the vertical direction. In the followingexplanation, an image having an angle of view covering 360° in thehorizontal direction and 360° in the vertical direction and that isobtained by photographing the range of a full sphere will be termed an“omnidirectional image”.

When the user employs the still image photographic function, the storageunit 23 stores a single omnidirectional image that has been captured bythe imaging unit 20 in a storage medium 51 (for example, a memory cardor the like). And, when the user employs the video photographicfunction, the storage unit 23 stores an omnidirectional video made upfrom a plurality of omnidirectional images that have been repeatedlycaptured by the imaging unit 20 in the storage medium 51. As mentionedabove, each of the frames of the omnidirectional video is anomnidirectional image. It should be understood that, although thestorage medium 51 in FIG. 2 can be inserted into and removed from theimaging device 2, it would also be possible for the imaging device 2 tofixedly incorporate the storage medium 51.

Explanation of the Omnidirectional Image

FIG. 3(a) is a schematic figure illustrating the image capturing rangeof the imaging unit 20. Taking the position of installation of theimaging unit 20 (i.e. the camera position) as the origin O, the imagingunit 20 captures an image over a full sphere 60 shown in FIG. 3(a). AndFIG. 3(b) is a schematic figure showing an example of an omnidirectionalimage captured by the imaging unit 20. The omnidirectional image 61shown by way of example in FIG. 3(b) includes a first hemisphericalimage 62 that has been captured by the first imaging element 201 and asecond hemispherical image 63 that has been captured by the secondimaging element 202. The first hemispherical image 62 includes acircular image 64 that is formed by the first image capturing opticalsystem 21. And the second hemispherical image 63 includes a circularimage 65 that is formed by the second image capturing optical system 22.

An image having any desired angle of view may be obtained by cutting outand deforming a portion of the omnidirectional image 61 shown by way ofexample in FIG. 3(b). For example, if it is desired to obtain an imagehaving an angle of view 66 shown in FIG. 3(a), then a region 67 of FIG.3(b) may be cut out and deformed into a rectangle. Moreover if, from theupper half hemisphere of the complete sphere as shown in FIG. 3(a), itis desired to obtain an image over a range 68 shown by the stippling inFIG. 3(a) that is described by carrying the line segment A-B around thesphere in the horizontal direction, then regions 69 of FIG. 3(b) may becut out and deformed into a rectangle. An example of an image 70obtained at this time is shown in FIG. 3(c). This image 70 is apanoramic landscape image. It should be understood that the left edge 71and the right edge 72 of the image 70 shown by way of example in FIG.3(c) are actually continuous with one another, as shown in FIG. 3(a). Inother words, the image 70 shown as an example in FIG. 3(c) is anall-around image captured over a range of 360° around the imaging unit20. This all-around image includes a path 600 that goes around thesurface of the full sphere 60. The path 600 is a circumference of acircle centered upon the origin O and having the same diameter as thediameter of the full sphere 60. Since the origin O is the center of thefull sphere 60, accordingly this circle coincides with the circumferenceof a cross section of the full sphere 60 sectioned by a plane passingthrough the center of the full sphere 60.

The length of the line segment A-B can be set arbitrarily. For example,by setting the point A to the so-called north pole and setting the pointB to the so-called south pole, the range in the omnidirectional image 61that is captured and the range in the all-around image 70 that iscaptured match one another. In other words, the all-around image may becalled a projection (i.e. a mapping) of the omnidirectional image 61onto a two dimensional image.

The image 70 shown as an example in FIG. 3(c) is an all-around imageobtained by capturing a range of 360° around the horizontal direction ofthe imaging unit 20. Accordingly, the all-around image 70 includes thepath 600 that corresponds to the so-called equator. This all-aroundimage is not limited to running around the imaging unit 20 in thehorizontal direction; it could be an image that is captured running overa range of 360° in any direction upon the imaging unit 20. For example,it would be possible for it to be an image that is captured running overa range of 360° around the imaging unit 20 along a meridian of the fullsphere 60.

In the following explanation, for simplicity of description, theomnidirectional image is shown as being an all-around image capturedover a range of 360° in the horizontal direction, by way of example, asshown in the example of FIG. 3(c). In other words, in the followingexplanation, the omnidirectional image is shown as though it were animage like the image 70 of FIG. 3(c), but actually the captured image isover the range of the full sphere 60 shown in FIG. 3(a).

It should be understood that the imaging unit 20 need not simply includetwo imaging elements, i.e. the first imaging element 201 and the secondimaging element 202; it could also be provided with a larger number ofimaging elements. By doing so, it would be possible to obtain anomnidirectional image even if a range over which each of the imagingelements is capable of capturing an image is more restricted than a fullhemisphere. In other words it would be arranged, not to capture an imageover the entire range of the full sphere 60 by combining the two imagingelements each of which captures an image over the range of a fullhemisphere, but rather to capture an image over the entire range of thefull sphere 60 by combining the three or more imaging elements each ofwhich captures an image over a more restricted range. It would also beacceptable for parts of the image capturing ranges of the variousimaging elements to overlap one another. For example, part of the imagecapturing range of the first imaging element 201 may overlap part of theimage capturing range of the second imaging element 202. In a similarmanner, the imaging device 2 need not only include two image capturingoptical systems, i.e. the first image capturing optical system 21 andthe second image capturing optical system 22; it would also beacceptable for it to be provided with a larger number of image capturingoptical systems, each of which forms an image of the photographicsubject over a more restricted range than a full hemisphere.

Furthermore, it would also be acceptable for the imaging unit 20 not toinclude two imaging elements, i.e. the first imaging element 201 and thesecond imaging element 202, but only a single imaging element. Forexample, by light from the first image capturing optical system 21 andlight from the second image capturing optical system 22 being directedto a single imaging element by a mirror or the like, the circular image64 and the circular image 65 may be captured by the single imagingelement. By doing this, it is possible to reduce the number of imagingelement components, so that a reduction in cost of the imaging unit 20may be anticipated.

Explanation of the Image Processing Device 3

FIG. 4(a) is a block diagram schematically showing the structure of theimage processing device 3. The image processing device 3 comprises animage generation unit 30, an input unit 31, and an output unit 32. Theinput unit 31 reads out an omnidirectional video from the storage medium51 in which the omnidirectional video is stored, and inputs it into theimage generation unit 30. The image generation unit 30 then performsprocessing for creation of a two dimensional video that will bedescribed hereinafter upon the omnidirectional video that has thus beeninputted. This processing to create a two dimensional video isprocessing that creates a two dimensional video from the omnidirectionalvideo. In other words, the image generation unit 30 creates a twodimensional video from the omnidirectional video that has been inputted.The two dimensional video is a video in which each frame is made of animage that has a more restricted angle of view than that of theomnidirectional image. For example, the two dimensional video may havecontent equivalent to that of a video that has been captured by placinga conventional video camera having an angle of view of around 50° to 25°at the origin O in FIG. 3(a). The output unit 32 stores the twodimensional video that has been created by the image generation unit 30in a storage medium 52. The storage medium 52 may be the same storagemedium as the storage medium 51 in which the imaging device 2 has storedthe omnidirectional video, or may be a different storage medium. Itshould be understood that although, in FIG. 4(a), the storage medium 51and the storage medium 52 are shown as being provided outside the imageprocessing device 3, it would also be acceptable for one or both of thestorage medium 51 and the storage medium 52 to be housed internally tothe image processing device 3. Moreover, the storage medium 51 and thestorage medium 52 could also be adapted to be connected to the imageprocessing device via a cable or wireless network. Alternatively,instead of the storage medium 51, it would also be possible to provide astructure in which the omnidirectional video is directly inputted fromthe imaging device 2 via a network.

Furthermore, each frame of a two dimensional video that has been createdfrom an omnidirectional video may include, not only one image with amore restricted angle of view than that of the omnidirectional image,but rather two or more images with more restricted angles of view thanthat of the omnidirectional image.

Explanation of the Reproduction Device 4

FIG. 4(b) is a block diagram schematically showing the structure of thereproduction device 4. The reproduction device 4 comprises a displayunit 40, an input unit 41, a control unit 42, and an actuation unit 43.The input unit 41 may read out an omnidirectional image from the storagemedium 51 in which the omnidirectional image is stored, and may input itto the control unit 42. The input unit 41 may read out a two dimensionalvideo from the storage medium 52 in which the two dimensional video isstored, and may input it to the control unit 42. And the control unit 42displays the omnidirectional image or the two dimensional video that hasthus been inputted upon the display unit 40. The display unit 40 may,for example, have a display screen that consists of a liquid crystalpanel or the like. And, on the basis of control by the control unit 42,the display unit 40 displays the omnidirectional image or the twodimensional video upon its display screen. It should be understood thatthe storage medium 51 and the storage medium 52 shown in FIG. 4(b) maybe provided externally to the reproduction device 4; but, alternatively,one or both of the storage medium 51 and the storage medium 52 may behoused internally to the reproduction device 4. The display unit 40, forexample, may be a liquid crystal display of a smart phone, may be aliquid crystal display of a tablet terminal, or may be a head mounteddisplay. Accordingly, if the entire region of an omnidirectional imageis displayed at once upon the display unit 40, the range of all round360° is being displayed on the two dimensional display so that it isdifficult for the user visually to confirm the image. Therefore, thereis a per se known method of reproduction in which part of theomnidirectional image having an angle of view of 360° is cut out, andonly a part of the omnidirectional image is displayed upon the twodimensional plane of the display screen (i.e. of the display unit 40).In the following explanation, this method of displaying and reproducingonly a part of the omnidirectional image upon the display unit 40 willbe assumed as a precondition.

The actuation unit 43 is an operation member via which actuations by theuser are inputted. In this embodiment, the actuation unit 43 is a touchsensor that is superimposed upon the display screen of the display unit40. The actuation unit 43 detects the position at which a finger of theuser or the like contacts the display screen, and transmits thisposition to the control unit 42. In other words, the actuation unit 43detects touch actuation by the user and inputs the result to the controlunit 42. Touch actuation may include, for example, scrolling operationin which the user contacts a finger or the like at a position upon thedisplay screen and slides this finger or the like in the upward,downward, leftward, or rightward direction while keeping it in thecontacted state and thereafter this finger or the like is no longercontacted to the display screen. In this embodiment, scrolling operationby the finger or the like moving leftward is termed “leftward scrollingoperation”.

Here, scrolling operation is an operation to shift the image displayedupon the display unit 40 in any desired direction upon the display unit40.

Moreover, it may be arranged for the actuation unit 43 to be some typeof actuation member other than a touch sensor. For example, if thereproduction device 4 is a head mounted display, then the user may alsoperform scrolling operation in the leftward direction by swinging his orher head leftward. In this case, the actuation unit 43 would be a sensorthat detects displacement of the head mounted display (i.e. itsorientation, its position, or the like) accompanying movement of thehead of the user. The amount by which the image displayed upon thedisplay unit 40 shifts corresponds to the amount of displacement of thehead mounted display. For example, the image displayed upon the displayunit 40 may be shifted rightward by the user performing the operation byswinging his or her head leftward.

It should be understood that the actuation member employed in theactuation unit 43 is not limited by those described above, provided thatthe image displayed upon the display unit 40 can be shifted in anydesired direction upon the display unit 40.

Now the processing for reproduction of the omnidirectional image by thereproduction device 4 (i.e. the display processing) will be explained.FIG. 5 shows figures for explanation of the reproduction processing forthe omnidirectional image. FIG. 5(a), FIG. 5(c), and FIG. 5(e) arefigures showing examples of an omnidirectional image 73 that is to bethe subject of reproduction. This omnidirectional image 73 is an imagein which a photographic subject 74 and a photographic subject 75 havebeen photographed. And FIG. 5(b), FIG. 5(d), and FIG. 5(f) are figuresshowing examples of screens displayed by the display unit 40 that isreproducing the omnidirectional image 73.

The control unit 42 cuts out a partial range 76 from the omnidirectionalimage 73 shown in FIG. 5(a) and displays it upon the display unit 40, asin FIG. 5(b). In FIG. 5(b), the display unit 40 is displaying the range76 that includes the photographic subject 74. And, when the userperforms leftward scrolling operation upon the screen, the control unit42 shifts the portion of the omnidirectional image 73 displayed upon thedisplay unit 40 in the leftward direction as shown in FIGS. 5(c) and5(d), and performs control to cause a portion of the omnidirectionalimage 73 to be displayed that was not being displayed upon the displayunit 40 at the time point 5(b). In other words, the control unit 42replaces the image displayed upon the display unit 40, and changes itfrom the portion of the omnidirectional image 73 that was beingdisplayed upon the display unit 40 to a different portion that is moreto the right side of the omnidirectional image 73. To put it in anothermanner, the control unit 42 temporarily deletes the portion of theomnidirectional image 73 that is currently being displayed upon thedisplay unit 40, and changes the range 76 shown in FIG. 5(a) to therange 77 shown in FIG. 5(c), so that a new portion of theomnidirectional image 73 that corresponds to the range 77 is nowdisplayed upon the display unit 40. At this time, from the point of viewof the user, it appears that the omnidirectional image 73 is shifted inthe leftward direction by a distance 78. To put it in another manner,from the point of view of the user, it appears that the image displayedupon the display unit 40 is shifted leftward by the distance 78. Here,the distance 78 can be measured by units of the pixels making up thedisplay unit 40. For example, by performing scrolling operation in theleftward direction of the screen by the minimum amount, theomnidirectional image 73 may be shifted in the leftward direction uponthe display unit 40 by one pixel. The distance 78 can be defined inunits of pixels by measuring by how many pixels the image has shifteduntil the range 77 is displayed upon the display unit 40. In thefollowing explanation, the change from the display state shown in FIG.5(a) and FIG. 5(b) to the display state shown in FIG. 5(c) and FIG. 5(d)will be expressed as “the image displayed upon the display unit 40 hasshifted by the distance 78 in the leftward direction”.

When the user repeatedly performs the scrolling operation in theleftward direction upon the screen, the control unit 42 repeats thecontrol described above. As a result, as shown in FIG. 5(e) and FIG.5(f), the control unit 42 comes to cut out a range 79 that includes thephotographic subject 75, and to display that range upon the display unit40. Since the left end and the right end of the omnidirectional image 73are mutually continuous as described above, accordingly, when the useragain repeats the leftward scrolling operation upon the screen, thecontrol unit 42 again comes to display the photographic subject 74 uponthe display unit 40. In other words, the contents displayed upon thedisplay unit 40 again come to be the contents shown in FIG. 5(a) andFIG. 5(b).

As described above, the omnidirectional image 73 is a portion of animage in which the photographic subject 74 and the photographic subject75 have been captured, and, is image data which is used, by repeatingcontrol to shift the portion of the omnidirectional image 73 that isdisplayed upon the display unit 40 in the leftward direction so as todisplay upon the display unit 40 portions of the omnidirectional image73 that have not yet been displayed, so that the photographic subject 75is displayed after the photographic subject 74 has been displayed, andthen the photographic subject 74 is displayed upon the display unit 40for a second time.

In this manner, a distance 80 (refer to (FIG. 5(e)) by which the imagedisplayed upon the display unit 40 is shifted until the photographicsubject 75 is displayed upon the display unit 40 by repeatedlyperforming the scrolling operation in the leftward direction upon thescreen after the photographic subject 74 is displayed upon the displayunit 40 is termed the distance from the photographic subject 74 to thephotographic subject 75 in relation to the leftward direction. In asimilar manner, a distance 81 (refer to (FIG. 5(e)) by which the imagedisplayed upon the display unit 40 is shifted until the photographicsubject 74 is displayed upon the display unit 40 by repeatedlyperforming the scrolling operation in the leftward direction upon thescreen after the photographic subject 75 is displayed upon the displayunit 40 is termed the distance from the photographic subject 75 to thephotographic subject 74 in relation to the leftward direction.

In a similar manner in the case in which, from the state shown in FIG.5(a) and FIG. 5(b), the user repeatedly performs scrolling operation,not in the leftward screen direction, but in the rightward screendirection, the photographic subject 75 is displayed after thephotographic subject 74 has been displayed, and then the photographicsubject 74 is again displayed for a second time upon the display unit40. However, in the example shown in FIG. 5(a) and FIG. 5(b), if thephotographic subject 75 is not positioned to the right of thephotographic subject 74 but is positioned upward therefrom, then byperforming scrolling operation, not in the leftward or rightwarddirection of the screen, but for example in the upward direction of thescreen or the like, after the photographic subject 74 has beendisplayed, the photographic subject 74 disappears from the screen, andthereafter the photographic subject 75 is displayed, and then thephotographic subject 74 comes to be displayed again for a second time.In other words, any two photographic subjects that have beenphotographed in the omnidirectional image 73 can be displayed asdescribed above by keeping the direction of scrolling operationconstant.

As described above, the reproduction device 4 of this embodiment cutsout a portion of the omnidirectional image having an angle of view of360° in the vertical direction and in the horizontal direction, andreproduces this portion upon the display screen which is a twodimensional plane. In the above explanation the omnidirectional image isdescribed as being a still image, but, by similar processing, it is alsopossible to reproduce an omnidirectional video of which each frame is anomnidirectional image. In this case, when the omnidirectional video ispaused and then is replayed again, the situation is effectively the sameas explained above. When an omnidirectional video is being reproduced,the only feature of difference is that the frames (i.e. theomnidirectional images) that make up the omnidirectional video changeover time.

When an omnidirectional image is reproduced, it is possible for the userto display the main photographic subject upon the display unit 40 and tocheck it visually by performing scrolling operation in any desireddirection. However since, when an omnidirectional video is replayed,each frame (i.e. each omnidirectional image) that makes up theomnidirectional video is only displayed upon the display unit 40 for anextremely short time interval, accordingly it is difficult to cause aportion that is not being displayed upon the display unit 40 of a givenframe to be displayed. Normally, scrolling operation results in thecontrol over the next frame to be displayed upon the display unit 40. Asa result, there is a possibility that the user may, for example,overlook a scene in which a main photographic subject is executing amovement that ought to be noticed in a portion that is not currentlybeing reproduced in the display screen. Furthermore, the user may noteven notice the existence of a main photographic subject which hehimself has not noticed in this omnidirectional video. Moreover, eachtime that the user views the video, it is necessary for him to adjustthe display position by performing scrolling operation as describedabove, which is quite troublesome. Again, if for example two mainphotographic subjects of interest are moving at separate locations, thenit is necessary for the user to replay the video several times in orderto see both of them. Due to this, the burden upon the user during replayof an omnidirectional video is quite high. Accordingly, the imageprocessing system 1 of this embodiment automatically creates from theomnidirectional video a two dimensional video that concentrates upon anappropriate photographic subject, and reproduces this two dimensionalvideo, thus solving the problem described above.

The reproduction processing (display processing) performed by thereproduction device 4 for a two dimensional video will now be explained.As described hereinafter, a two dimensional video is built up from aplurality of two dimensional images that are arranged in time series.Each two dimensional image that makes up the two dimensional video istermed a “frame”. The control unit 42 reproduces the two dimensionalvideo by displaying this plurality of frames in sequence upon thedisplay unit 40.

It should be understood that input of the omnidirectional video from theimaging device 2 to the image processing device 3 may also be performedby a method that does not employ the storage medium 51. For example, itmay be arranged for the imaging device 2 and the image processing device3 to be electrically connected together by a communication cable, andfor the omnidirectional video be inputted to the image processing device3 by data communication. Alternatively, it may be arranged for theomnidirectional video to be transferred between the imaging device 2 andthe image processing device 3 by wireless communication via radio waves.The same is the case for input of the omnidirectional image from theimaging device 2 to the reproduction device 4, and for input of the twodimensional video from the image processing device 3 to the reproductiondevice 4.

Explanation of Processing to Create a Two Dimensional Video

The processing performed by the image generation unit 30 for creating atwo dimensional video will now be explained. The image generation unit30 creates a two dimensional video from the omnidirectional video byexecuting two dimensional video creation processing. This twodimensional video creation processing is processing for specifying amain photographic subject from the omnidirectional image, and forcreating a two dimensional video that includes this main photographicsubject that has thus been specified.

The two dimensional video creation processing includes processing tospecify the photographic subject and processing to create a twodimensional image. The photographic subject specification processing isprocessing to specify a main photographic subject from anomnidirectional image included in the omnidirectional video. And theprocessing to create a two dimensional image is processing to create atwo dimensional image including the main photographic subject asspecified by the photographic subject specification processing from theomnidirectional image. The photographic subject specification processingand the processing to create a two dimensional image will be explainedin the following in order.

Explanation of the Photographic Subject Specification Processing

The image generation unit 30 specifies a main photographic subject fromeach frame included in a single omnidirectional video by employing a perse known technique such as facial recognition or pattern matching or thelike. For example, if the main photographic subject is a person, then itis possible to detect a face included in the omnidirectional image byemploying a technique for recognizing the face of a human being, and tospecify the whole body of the person corresponding to the detected facefrom its orientation, position, color, and so on. It should beunderstood that “to specify the main photographic subject” means torecognize (i.e. to detect) the positions and the shapes of variousphotographic subjects that appear in the omnidirectional image, and alsoto select a main photographic subject from among those photographicsubjects. For example, if the main photographic subject is a person andthree or more people have been detected from the omnidirectional image,then the image generation unit 30 may specify all of those persons asbeing the main photographic subject.

Recognition of the main photographic subject may be determined on thebasis of various factors (parameters) such as the size, saliency of thephotographic subject in the image or the like. Moreover, suchrecognition may be determined on the basis of movement of thephotographic subject or the like by employing a plurality of images thatare temporally continuous, instead of employing only one image. Itshould be understood that, by expressing a parameter numerically and byemploying threshold value processing, it is possible to take aphotographic subject for which some parameter exceeds a predeterminedthreshold value as being the main photographic subject. When thresholdvalue processing is employed, a plurality of photographic subjects maybe recognized as being the main photographic subject. There may be onlyone main photographic subject, or a plurality thereof. It should beunderstood that, since a range of 360° is captured in theomnidirectional image, accordingly the possibility is high that aplurality of photographic subjects will be recognized as being the mainphotographic subjects, as compared with the case of photography with anormal camera.

Explanation of the Processing to Create a Two Dimensional Video

The processing to create a two dimensional video is processing to createa two dimensional image including the main photographic subject fromeach frame of the omnidirectional video. The image processing system 1of this embodiment is configured to automatically create a twodimensional video including a main photographic subject from anomnidirectional video. Each image making up the two dimensional video istermed a “frame”. The processing to create a two dimensional video isprocessing to create two dimensional images (i.e. frames) including themain photographic subject specified by the photographic subjectspecification processing from the omnidirectional video. In thisprocessing to create two dimensional images, if there is only one mainphotographic subject, then a frame that includes this one mainphotographic subject is generated, whereas, if there are two mainphotographic subjects, then a frame including these two mainphotographic subjects is generated.

FIGS. 7(a) through 7(c) are figures showing examples of two dimensionalimages (i.e. frames) that are generated by the processing to create atwo dimensional image when two main photographic subjects have beenrecognized. A first photographic subject 201 and a second photographicsubject 202 are photographic subjects that have been recognized as beingmain photographic subjects. A two dimensional image 610 shown by way ofexample in FIG. 7(a) is a two dimensional image (i.e. a frame) obtainedby cutting out a partial image (i.e. an angle of view) that includes thefirst photographic subject 201 and the second photographic subject 202from the omnidirectional image. Furthermore, as shown in FIG. 7(b), itwould also be acceptable to create a two dimensional image 611 bycutting out the first photographic subject 201 from the omnidirectionalimage, cutting out the second photographic subject 202 from theomnidirectional image, and sticking these two cutout partial imagestogether in the upward, downward, leftward, or rightward direction.Moreover, as shown in FIG. 7(c), it would also be acceptable to create asynthesized two dimensional image 612 by superposing a partial image 613in which the first photographic subject 201 has been cut out from theomnidirectional image upon an image in which a wide range including thesecond photographic subject 202 has been cut out from theomnidirectional image.

In the following, a problem with the processing to create a twodimensional image will be explained by employing an example in which avolleyball match is imaged by the imaging device 2.

FIG. 6 shows figures for explanation of the processing to create a twodimensional image. FIG. 6(a) is a plan view of a volleyball court. Inthe example of FIG. 6, the imaging device 2 is installed in the centerof the court 200. A person who is a main photographic subject(hereinafter termed the “first photographic subject 201”) is present onthe left side of the court 200 in the drawing. And another person who isanother main photographic subject (hereinafter termed the “secondphotographic subject 202”) is present on the right side of the court 200in the drawing. In other words, this is a case in which two mainphotographic subjects have been recognized. And FIG. 6(b) shows thearrangement of the first photographic subject 201 and the secondphotographic subject 202 in three dimensional space, centered upon theimaging device 2.

Here, let us consider a case such as in FIG. 7(a) in which a twodimensional image (i.e. a frame) that includes the first photographicsubject 201 and the second photographic subject 202 has been cut outfrom the omnidirectional image. If it is supposed that the imagegeneration unit 30 generates a two dimensional image (i.e. a frame) soas to include the path 204 of FIG. 6(b), then the two dimensional image610 that is generated has the first photographic subject 201 disposedupon its right side and the second photographic subject 202 disposedupon its left side. On the other hand, if it is supposed that the imagegeneration unit 30 generates a two dimensional image (i.e. a frame) soas to include the path 209 of FIG. 6(b), then the two dimensional image610 that is generated has the first photographic subject 201 disposedupon its left side and the second photographic subject 202 disposed uponits right side. The problem of arrangement described above also occursin a similar manner for the two dimensional image 611 and for the twodimensional image 612. In other words, due to the nature of theomnidirectional image, the image generation unit 30 can arrange thefirst photographic subject 201 and the second photographic subject 202in at least two ways (i.e. an arrangement with the first photographicsubject 201 on the left side and the second photographic subject 202 onthe right side, or in an arrangement with the first photographic subject201 on the right side and the second photographic subject 202 on theleft side). It should be understood that, for ease of understanding, thepath 204 shown in FIG. 6(b) is illustrated in FIG. 7(a).

Now, when the two dimensional image (or the two dimensional video)generated by the image generation unit 30 is replayed, the visibilityfor the user differs greatly according to the manner of arrangement. Forexample, with this example of volleyball, if it is supposed that thefirst photographic subject 201 is the receiver and the secondphotographic subject 202 is the attacker, then an image (i.e. a video)that imparts a sense of discomfort results from the arrangement of thefirst photographic subject 201 and the second photographic subject 202due to the relationship with the volleyball captured in the twodimensional image 610. Accordingly, it is necessary for the imagegeneration unit 30 to generate a two dimensional image (i.e. a frame) inwhich the plurality of main photographic subjects (i.e. the two mainphotographic subjects) are arranged in an appropriate manner.

In processing to create a two dimensional image, the image generationunit 30 of this embodiment creates a two dimensional image that includesboth the first photographic subject 201 and the second photographicsubject 202 as the main photographic subjects. Using the positions ofthe first photographic subject 201 and the second photographic subject202 that have been specified by the photographic subject specificationprocessing, the image generation unit 30 determines the angle of viewincluding the first photographic subject 201 and the second photographicsubject 202 as being the angle of view of the two dimensional image. Forexample, if the first photographic subject 201 and the secondphotographic subject 202 are present in the positions shown in FIG.6(a), then the image generation unit 30 determines an angle of view 203including the first photographic subject 201 and the second photographicsubject 202 as being the angle of view of the two dimensional image. Andthe image generation unit 30 creates a two dimensional image by cuttingout contents corresponding to the angle of view 203 from theomnidirectional image, and deforming these contents into a rectangle.

There are a plurality of possible angles of view that include the firstphotographic subject 201 and the second photographic subject 202. Forexample, there is an angle of view in which the first photographicsubject 201 is disposed on the left of the screen and the secondphotographic subject 202 is disposed on the right of the screen, andthere is also an angle of view in which the first photographic subject201 is disposed on the right of the screen and the second photographicsubject 202 is disposed on the left of the screen. From among a numberof angles of view of this sort, the image generation unit 30 selects theangle of view that “includes the shortest path 204 that connects thefirst photographic subject 201 and the second photographic subject 202in three dimensional space, and that also includes the firstphotographic subject 201 and the second photographic subject 202”. Forexample, in FIG. 6(a), as angles of view that include the firstphotographic subject 201 and the second photographic subject 202, anumber of angles of view may be considered, such as the angle of view203, an angle of view 205 and so on. From among those angles of view,the image generation unit 30 selects the angle of view 203 that includesthe shortest path 204 that connects the first photographic subject 201and the second photographic subject 202, and that also includes both thefirst photographic subject 201 and the second photographic subject 202.It should be understood that “includes the shortest path 204” also canbe considered as meaning “includes a third photographic subject that ispresent in the shortest path 204 and that is different from the firstphotographic subject 201 and the second photographic subject 202”.

A method for specifying the “shortest path connecting the firstphotographic subject 201 and the second photographic subject 202 in theomnidirectional image” will now be explained. When the full sphere 60 issectioned by a plane that passes through the center of the full sphere60 and also passes through the first photographic subject 201 andthrough the second photographic subject 202, a part of the circumferenceof this cross section of the full sphere 60 is the shortest path thatconnects the first photographic subject 201 and the second photographicsubject 202. In the omnidirectional image 206 shown in FIG. 6(b), if thefirst photographic subject 201 and the second photographic subject 202are regarded as being points, then the circumference of the crosssection of the full sphere 60 may be considered to be constituted byconnecting the path 204 and the path 209 together. The shorter one amongthe path 204 and the path 209 is the shortest path. In other words, theshortest path connecting the first photographic subject 201 and thesecond photographic subject 202 in the omnidirectional image is the path204. The shortest path can be specified uniquely, except when the firstphotographic subject 201 and the second photographic subject 202 are atdirectly opposite positions upon the full sphere 60.

The image generation unit 30 calculates the shortest path between thefirst photographic subject 201 and the second photographic subject 202in the following manner. For example, with the omnidirectional image(i.e. the all-around image) 206 shown in FIG. 6(c), the image generationunit 30 arranges the second photographic subject 202 in the rightwarddirection from the first photographic subject 201. Alternatively, theimage generation unit 30 may prepare an omnidirectional image 206 inwhich the second photographic subject 202 is arranged in the rightwarddirection from the first photographic subject 201. As a result, thestraight line that connects the first photographic subject 201 and thesecond photographic subject 202 in the omnidirectional image 206coincides with the circumference of a cross section of the full sphere60 when the full sphere 60 is sectioned by a plane that passes throughthe center of the full sphere 60 and also passes through the firstphotographic subject 201 and the second photographic subject 202. Theimage generation unit 30 compares together the path 209 from the secondphotographic subject 202 to the first photographic subject 201 in theleftward direction, and the shortest path 204 from the firstphotographic subject 201 to the second photographic subject 202 in theleftward direction. It should be understood that, in this comparison,the image generation unit 30 calculates a distance 208 (hereinafterreferred to as the first distance 208) from the first photographicsubject 201 to the second photographic subject 202 in the leftwarddirection. In a similar manner, the image generation unit 30 calculatesa distance 207 (hereinafter referred to as the second distance 207) fromthe second photographic subject 202 to the first photographic subject201 in the leftward direction. The distance may be calculated bycounting the pixels that constitute the omnidirectional image (theall-around image) 206. And the image generation unit 30 comparestogether the first distance 208 and the second distance 207. In theexample of FIG. 6, the second distance 207 is longer than the firstdistance 208.

Next, a method for generation of a two dimensional image (i.e. a frame)by the image generation unit 30 will be explained. The image generationunit 30 compares together the first distance 208 and the second distance207 as described above, and determines that the second distance 207 islonger than the first distance 208. Accordingly, the image generationunit 30 generates a two dimensional image (i.e. a frame) in which thefirst photographic subject 201 is arranged on the right and the secondphotographic subject 202 is arranged on the left. On the other hand, ifthe first distance 208 is longer than the second distance 207, then,conversely, the image generation unit 30 generates a two dimensionalimage (i.e. a frame) in which the first photographic subject 201 isarranged on the left and the second photographic subject 202 is arrangedon the right.

Moreover, it would also be possible for the image generation unit 30 togenerate a two dimensional image (i.e. a frame) in the following manner.When the image generation unit 30 compares together the first distance208 and the second distance 207, if it has been determined that thesecond distance 207 is longer than the first distance 208, then theangle of view is determined so as to include the shortest path 204, andmoreover so as to include the first photographic subject 201 and thesecond photographic subject 202. And, when a partial image is cut outfrom the omnidirectional image (i.e. the all-around image) 206 with theangle of view that has thus been determined, the first photographicsubject 201 is arranged on the right side of the second photographicsubject 202 in the partial image. Accordingly, the image generation unit30 generates a two dimensional image (i.e. a frame) so that the firstphotographic subject 201 is arranged on the right and the secondphotographic subject 202 is arranged on the left.

It should be understood that the first distance 207 described abovecorresponds to a distance by which the image displayed upon the displayunit 40 shifts when the user repeats the scrolling operation in theleftward direction, after the first photographic subject 201 has beendisplayed upon the display unit 40 until the second photographic subject202 is displayed upon the display unit 40, in the reproductionprocessing for the omnidirectional image by the reproduction device 4described above. And the second distance 208 described above correspondsto a distance by which the image displayed upon the display unit 40shifts when the user repeats the scrolling operation in the leftwarddirection, after the second photographic subject 202 has been displayedupon the display unit 40 until the first photographic subject 201 isdisplayed upon the display unit 40, in the reproduction processing forthe omnidirectional image by the reproduction device 4 described above.

As described above, the image generation unit 30 generates, from theomnidirectional image, a two dimensional image in which the firstphotographic subject 201 and the second photographic subject 202 arearranged, on the basis of the first distance 207 by which the imagedisplayed upon the display unit 40 shifts from when the firstphotographic subject 201 is displayed upon the display unit 40 until thesecond photographic subject 202 is displayed upon the display unit 40,and on the basis of the second distance 208 by which the image displayedupon the display unit 40 shifts from when the second photographicsubject 202 is displayed upon the display unit 40 until the firstphotographic subject 201 is displayed upon the display unit 40. Inconcrete terms, if the first distance 207 is longer than the seconddistance 208, then the image generation unit 30 generates a twodimensional image from the omnidirectional image which includes thefirst photographic subject 201 and the second photographic subject 202,and in which the second photographic subject 202 is disposed on the leftside of the first photographic subject 201.

What direction the left side (i.e. the first direction side) of thefirst photographic subject 201 indicates will now be described in detailby using FIG. 7(d). It should be understood that, in FIG. 7(d), theposition 201 a of the first photographic subject 201 and the position202 a of the second photographic subject 202 are represented by dots forthe sake of simplicity of explanation. When the position 201 a of thefirst photographic subject 201 and the position 202 a of the secondphotographic subject 202 are known, it is possible to obtain a vector615 having the first photographic subject 201 as start point and thesecond photographic subject 202 as end point. This vector 615 may bedecomposed into a component 616 in the leftward direction (i.e. in thehorizontal direction) and a component 617 in the direction orthogonal tothe leftward direction (i.e. in the vertical direction). If thecomponent 616 in the leftward direction is positive in terms of theleftward direction, then the second photographic subject 202 ispositioned on the left side of the first photographic subject 201. Inother words, “the second photographic subject 202 is disposed on theleft side (i.e. on the first direction side) of the first photographicsubject 201” means that, the vector 615 whose start point is theposition 201 a of the first photographic subject 201 and whose end pointis the position 202 a of the second photographic subject 202 has apositive component in the leftward direction (i.e. in the firstdirection). It should be noted that it does not matter what the state ofthe component 617 of this vector 615 in the direction (i.e. the verticaldirection) orthogonal to the leftward direction (i.e. the firstdirection) is.

It should be noted that, for generating the two dimensional image (i.e.the frame) with the image generation unit 30, while an idea of using thefirst distance 207 and the second distance 208 has been explained withreference to FIG. 6, another method can also be explained by employing,not distances, but rather angles in three dimensional space. Forexample, in FIG. 6(b), consider the angle formed by a vector from theorigin O toward the first photographic subject 201, and a vector fromthe origin O toward the second subject 202. There are two angles formedby these vectors to be considered: an acute angle and an obtuse angle.Among these, the acute angle corresponds to the shortest path 204 and tothe angle of view 203, while the obtuse angle corresponds to the path209 and to the angle of view 205. Accordingly, the image generation unit30 determines the angle of view so that the angle formed by these twovectors becomes minimum, and moreover so as to include the firstphotographic subject 201 and the second photographic subject 202. When apartial image is cut out from the omnidirectional image (i.e. theall-around image) 206 with the angle of view that has thus beendetermined, the first photographic subject 201 is arranged on the rightside of the second photographic subject 202 in the partial image.Accordingly, the image generation unit 30 generates a two dimensionalimage (i.e. a frame) so that the first photographic subject 201 isarranged on the right and the second photographic subject 202 isarranged on the left.

The image generation unit 30 generates (i.e., creates) two dimensionalimages (i.e., frames) by the processing explained above. And the imagegeneration unit 30 generates (i.e. creates) a two dimensional video thatincludes those two dimensional images, and stores it upon the storagemedium 52.

FIG. 8 is a flow chart for the two dimensional video creationprocessing. First in step S10 the image generation unit 30 performs thephotographic subject specification processing upon each frame includedin the omnidirectional video. Due to this, the main photographic subjectfor each frame is specified.

Then in step S30 the image generation unit 30 selects one frame includedin the omnidirectional video. And the image generation unit 30 acquiresthe number of main photographic subjects that are specified in thisselected frame. If there is only a single main photographic subject (YESin step S30), then the flow of control is transferred to step S35. Instep S35, on the basis of the omnidirectional image (i.e., frame), theimage generation unit 30 creates a two dimensional image (i.e. a frame)that includes the main photographic subject.

But if there are two or more main photographic subjects (NO in stepS30), then the flow of control proceeds to step S40. In step S40, theimage generation unit 30 calculates the first distance in the selectedframe. In other words, the image generation unit 30 takes one of the twomain photographic subjects as being the first photographic subject andthe other as being the second photographic subject, and calculates thedistance from the first photographic subject along a first directionuntil arriving at the second photographic subject. Then in step S50 theimage generation unit 30 calculates the second distance in the selectedframe. In other words, the image generation unit 30 calculates thedistance from the second photographic subject along the first directionuntil arriving at the first photographic subject. It should beunderstood that by the first direction is meant a direction in which,when a part of a frame included in the omnidirectional video is beingdisplayed upon the display unit 40, and when the user repeats thescrolling operation in some direction, after the first photographicsubject 201 has been displayed upon the display unit 40, the firstphotographic subject 201 disappears from the display unit 40, andthereafter the second photographic subject 202 is displayed, and thenthe first photographic subject 201 is again displayed upon the displayunit 40 for a second time.

In step S60, the image generation unit 30 determines whether or not thefirst distance calculated in step S40 is longer than the second distancecalculated in step S50. If the first distance is longer than the seconddistance, then the image generation unit 30 transfers the flow ofcontrol to step S70. In step S70, on the basis of the omnidirectionalimage (i.e. the frame) selected in step S30, the image generation unit30 creates a two dimensional image in which the second photographicsubject is arranged towards the first direction from the firstphotographic subject. On the other hand, if the first distance is lessthan or equal to the second distance, then the image generation unit 30transfers the flow of control to step S80. In step S80, on the basis ofthe frame selected in step S30, the image generation unit 30 creates atwo dimensional image in which the first photographic subject isarranged towards the first direction from the second photographicsubject.

Then in step S90 the image generation unit 30 determines whether or notany frame that has not yet been selected remains in the omnidirectionalvideo. If some frame remains that has not yet been selected, then theimage generation unit 30 transfers the flow of control to step S30. Onthe other hand, if all of the frames have already been selected, thenthe image generation unit 30 transfers the flow of control to step S100.In step S100, the image generation unit 30 controls the output unit 32so as to store the two dimensional video made up from the twodimensional images that have been created in steps S70 and S80 upon thestorage medium 52.

According to the embodiment described above, the following advantageousoperational effect is obtained.

(1) According to the structure of this embodiment, it is possibleautomatically to create a two dimensional image that is suitable forviewing from the omnidirectional image.

Variant of First Embodiment

It should be understood that a single device may incorporate two or moreamong the imaging unit 20, the image generation unit 30, and the displayunit 40. For example, the imaging device 2 may incorporate the imagegeneration unit 30 in addition to the imaging unit 20. In this case, theimaging device 2 would also fulfil the role of the image processingdevice 3. Accordingly, the image processing device 3 may not be includedin the image processing system 1. And, as another example, the imageprocessing device 3 may incorporate the display unit 40 in addition tothe image generation unit 30. In this case, the image processing device3 would also fulfil the role of the reproduction device 4. Accordingly,the reproduction device 4 may not be included in the image processingsystem 1. As another example, in addition to the imaging unit 20, theimaging device 2 may also include the image generation unit 30 and thedisplay unit 40. In this case, the imaging device 2 would also fulfilthe roles of the image processing device 3 and the reproduction device4. In other words, the imaging device 2 would, by itself, providefunctions equivalent to those of the image processing system 1.

FIG. 10 is a block diagram schematically showing an electronic device1000 that combines the image processing device 3 and the reproductiondevice 4. This electronic device 1000 may be, for example, a smart phoneor a tablet terminal. The electronic device 1000 comprises an imagegeneration unit 30, an input unit 31, an output unit 32, a display unit40, a control unit 42, and an actuation unit 43. The electronic device1000 is capable of creating a two dimensional video, reproducing uponthe display unit 40 a two dimensional video that has been created,storing upon the storage medium 52 a two dimensional video that has beencreated, and performing reproduction of an omnidirectional image (i.e.an omnidirectional video) upon the display unit 40. It should beunderstood that the operation of each of these sections of theelectronic device 1000 is the same as in the case of the firstembodiment, and accordingly explanation thereof will be omitted.

According to the variant embodiment described above, the followingadvantageous operational effect is obtained.

(2) According to the structure of this embodiment, a similaradvantageous effect to that of the embodiment described above can beobtained.

It should be understood that the creation of a two dimensional video bythe image generation unit 30 could be performed in real time in parallelwith the creation of an omnidirectional video by the imaging unit 20, orcould be started after the creation of the omnidirectional video hasbeen completed. In a similar manner, the display of the two dimensionalvideo by the display unit 40 may be performed in real time in parallelwith the creation of a two dimensional video by the image generationunit 30, or could be started after the creation of the two dimensionalvideo has been completed.

In the embodiment described above, it was explained that the imagingunit 20 captures an image of a full sphere. In other words, although ithas been explained that the imaging unit 20 is capable of capturing animage over a full range of 360° around itself, it would also be possiblefor the imaging unit 20 only to be capable of capturing an image over anarrower range than a full sphere, in the vertical direction and/or inthe horizontal direction. For example, the imaging unit 20 may beadapted to capture an image over a hemisphere. Alternatively, theimaging unit 20 may only be capable of capturing an image over a rangethat is yet more restricted than a hemisphere. For example, the imagingunit may only be capable of capturing an image over the range 68 shownby stippling in FIG. 3(a). If the angle of view of the imaging unit 20is more restricted than a full sphere, then the two dimensional videowill be composed of images whose angles of view are yet more restricted.

Furthermore, the all-around image may not necessarily be an image thatis captured over the entire range of 360°. For example, it would also bepossible to treat an image that has been captured over a range of around300° as an all-around image in which the left and right ends areconnected together. The same is the case for an omnidirectional image;it would also be possible to treat an image of which a portion of thecomplete sphere is missing as being an omnidirectional image which ismade continuous everywhere.

In this specification, the omnidirectional image is an image in which,by repeating control in which a part of an image that is displayed uponthe display unit 40 is shifted in the first direction so that a portionof the image that is not displayed upon the display unit 40 isdisplayed, after the first photographic subject 201 which is included inthat image is displayed, the second photographic subject 202 isdisplayed, and then the first photographic subject 201 is againdisplayed upon the display unit 40 for a second time. Moreover, an imagewith a part of the full sphere missing is also an omnidirectional imagesince, by making the missing portion continuous by connecting over itand by repeating control to shift the part of the image displayed uponthe display unit 40 in the first direction and to display the part ofthe image that is not displayed upon the display unit 40, after thefirst photographic subject 201 that is included in that image isdisplayed, the second photographic subject 202 is displayed, and thenthe first photographic subject 201 is again displayed upon the displayunit 40 for a second time.

FIG. 9(a) is a schematic figure showing an example of an omnidirectionalimage. An image 620 and an image 621 that correspond to hemispheres areimages that have been obtained by capturing over smaller ranges than360°, so that portions of the hemispheres are missing. The imagegeneration unit 30 and the control unit 42 are able to handle theseimages by treating the side E-F and the side G-F as being mutuallycontinuous. In other words, the images shown in FIG. 9(a) areomnidirectional images.

FIG. 9(b) is a schematic figure showing an example of an all-aroundimage. In FIG. 9(b), an image 622, an image 623, and an image 624 thathave been captured by imaging over discontinuous ranges in thehorizontal direction are illustrated. These three images are images thathave been captured over ranges that are smaller than 360° in thehorizontal direction, so that a portion of the full 360° extent ismissing. The image generation unit 30 and the control unit 42 are ableto treat the sides A-B, the sides C-D, and the sides E-F as though theyare continuous. In concrete terms, in the state in which a portion ofthe image 622 is being displayed upon the display unit 40, the controlunit 42 repeatedly performs control to cause the image 622 to be shiftedin the horizontal leftward direction, so that another portion of theimage 622 that is not displayed upon the display unit 40 is now causedto be displayed. As a result, the sides C-D are displayed, andsubsequently the sides E-F are displayed. And, by further repeating thiscontrol, the sides A-B are displayed, and the system returns to thestate in which the portion of the image 622 is displayed upon thedisplay unit 40. Accordingly, the image 622, the image 623, and theimage 624 shown in FIG. 9(b) constitute an all-around image. This isbecause the image shown in FIG. 9(b) (i.e. the image 622, the image 623,and the image 624) is an image in which, by taking appropriatephotographic subjects included in that image as being the firstphotographic subject 201 and the second photographic subject 202, and byrepeatedly performing control so as to cause a part of the imagedisplayed upon the display unit 40 to be shifted in the first directionand another portion of the image that is not being displayed upon thedisplay unit 40 now to be displayed, after the first photographicsubject 201 included in that image has been displayed, the secondphotographic subject is displayed, and then again the first photographicsubject 201 is displayed for a second time. It should be understood thathere, by an “all-around image” (or a “full sphere image”), is meant animage in which the continuity of the image contents is of no importance.In other words, for example, when the image that includes the sides C-Dis displayed upon the display unit 40, the user may visually confirmthat the image contents on the left of the side C-D and the imagecontents on the right of the side C-D are mutually discontinuous.However, continuity of the image contents does not matter, butcontinuity of the images is important. In other words, it will sufficeif the image on the left of the side C-D and the image on the right ofthe side C-D are continuous with one another.

For example, first considering the image 622, this image 622 may betermed an all-around image, by treating the side A-B and the side C-D ascontinuous. Moreover, considering the image 623, this image 623 may betermed an all-around image, by treating the side C-D and the side E-F ascontinuous. All of the images are omnidirectional images (all-aroundimages), since they are images for which, by treating all of them in asimilar manner, and by repeating control to cause a portion of the imagedisplayed upon the display unit 40 to shift in the first direction sothat another portion of the image that is not displayed upon the displayunit 40 is now displayed, after the first photographic subject 201included in that image is displayed the second photographic subject 202is displayed, and then the first photographic subject 201 is againdisplayed upon the display unit 40 for a second time.

It should be understood that, as shown in FIG. 9(b), the processingperformed by the image generation unit 30 for generation of a twodimensional image is in no way different from the embodiment describedabove, even for the all-around image and the omnidirectional image(i.e., the image 622, the image 623, and the image 624) which are notactually mutually continuous. In other words, how to generate a twodimensional image (i.e. a frame) in which the first photographic subject201 and the second photographic subject 202 are arranged from theall-around image (the image 622, the image 623, and the image 624) maybe determined in a similar manner to the case with the embodimentdescribed above. For example, the distance through which the imagedisplayed upon the display unit 40 is shifted by repeatedly performingoperation to scroll the screen in the leftward direction from when thefirst photographic subject 201 is displayed upon the display unit 40until the second photographic subject 202 is displayed upon the displayunit 40, and the distance through which the image displayed upon thedisplay unit 40 is shifted by repeatedly performing operation to scrollthe screen in the leftward direction from when the second photographicsubject 202 is displayed upon the display unit 40 until the firstphotographic subject 201 is displayed upon the display unit 40, may becompared together, and, if the former distance is longer than the latterdistance, then a two dimensional image may be created so that the secondphotographic subject 202 is arranged in the leftward direction from thefirst photographic subject 201.

According to the variant embodiment described above, the followingadvantageous operational effect is obtained.

(3) According to the structure of this embodiment, it is possibleautomatically to generate a two dimensional image that is suitable forviewing from the all-around image.

It should be understood that it would also be possible to create the twodimensional image (i.e. the frame) that includes the first photographicsubject 201 and the second photographic subject 202 by some method otherthan that described above with reference to FIG. 7(a) through FIG. 7(c).For example, a two dimensional image in which the space between thefirst photographic subject 201 and the second photographic subject 202is compressed may be created by employing a technique such as seamcarving or the like. Alternatively, it would also be possible to createa two dimensional image by decimating or shrinking a photographicsubject present between the first photographic subject 201 and thesecond photographic subject 202.

It would also be possible to arrange for the image generation unit 30not to perform the photographic subject specification processing for allof the frames, but rather for only some of the frames. For example, theimage generation unit 30 may specify the main photographic subject forevery 30th frame, i.e. for the first frame, the 31st frame, the 61stframe, and so on. Thus, the image generation unit 30 does not executethe photographic subject specification processing for the 29 framesbetween the first frame and the 31st frame.

For example, if the frame rate of the omnidirectional video is 60 fps,then 30 frames corresponds to 0.5 seconds. During a period of around 0.5seconds, it may be expected that the position of the main photographicsubject hardly changes. In other words, the position of the mainphotographic subject in the above described 29 frames can be easilyestimated from the position of the main photographic subject in thefirst frame and the position of the main photographic subject in the31st frame.

In this manner, by executing the photographic subject specificationprocessing only for some of the omnidirectional images (i.e. byspecifying the main photographic subject from only some of theomnidirectional images), it is possible to reduce the amount ofcalculation required for performing the two dimensional video creationprocessing.

Second Embodiment

In an image processing system according to the second embodiment, thedetails of the two dimensional video creation processing performed bythe image generation unit 30 are different from the case of the firstembodiment. It should be understood that features that are not mentionedin connection with this second embodiment are the same as the detailsexplained in connection with the first embodiment. In other words, thedetails explained in connection with the first embodiment are all to beconsidered as being incorporated in this second embodiment. In thefollowing, the image processing system according to the secondembodiment will be explained with emphasis centering upon the featuresof difference from the image processing system of the first embodiment.

In a similar manner to the case with the first embodiment, the imagegeneration unit 30 performs the photographic subject specificationprocessing for each frame. The image generation unit 30 performsorientation specification processing for the main photographic subjectthat has been specified, in which the orientation of the mainphotographic subject within the frame is specified. In this embodimentthe main photographic subject is a person, and by the orientation of themain photographic subject is meant the orientation of the face of theperson within the image. In the orientation specification processing,the image generation unit 30 performs per se known facial recognitionprocessing, and thereby recognizes the face of the main photographicsubject and the orientation of that face. And the image generation unit30 specifies the orientation of the face of the main photographicsubject within the image as being the orientation of this mainphotographic subject.

Next, a method for specifying the orientation of the main photographicsubject in the image will be explained. First, the orientation of themain photographic subject in three dimensional space is determined. Forexample, if the main photographic subject is a human being, then thedirection in which his or her nose is pointing is taken as being his orher orientation. In this case, the orientation of the vector whose startpoint is the center of his or her face and whose end point is the apexof his or her nose may be taken as being the orientation of the mainphotographic subject. The method for determining the orientation of themain photographic subject in three dimensional space will be describedhereinafter. When a vector giving the orientation of the mainphotographic subject in three dimensional space is specified, thatvector is projected onto the image (or onto the imaging surface). As aresult, the vector projected onto the two dimensional image in which themain photographic subject is imaged (i.e. the projected vector) becomesthe orientation of the main photographic subject in the image.

FIG. 11 is a figure for explanation of the processing to create a twodimensional image. The omnidirectional image 300 shown in FIG. 11includes a first photographic subject 301 and a second photographicsubject 302, which are the main photographic subjects. The distance, dueto control to shift the portion of the omnidirectional image 300displayed upon the display unit 400 in the leftward direction, by whichthe image displayed upon the display unit 40 is shifted from the firstphotographic subject being displayed upon the display unit 40 until thesecond photographic subject is displayed upon the display unit 40 (i.e.the distance in the rightward direction from the first photographicsubject 301 to the second photographic subject 302) is longer than thedistance from the second photographic subject 302 in the rightwarddirection to the first photographic subject 301. Accordingly, if it issupposed that a two dimensional image is created by performingprocessing similar to that of the first embodiment, then a twodimensional image is created in which the second photographic subject302 is arranged on the left side of the first photographic subject 301.

As described above, the nature of the omnidirectional image is such thatthe image generation unit 30 is capable of arranging the firstphotographic subject 201 and the second photographic subject 202 in atleast two ways (i.e., with the first photographic subject 201 on theleft side and the second photographic subject 202 on the right side, orwith the first photographic subject 201 on the right side and the secondphotographic subject 202 on the left side). On the other hand, when atwo dimensional image (or a two dimensional video) generated by theimage generation unit 30 is reproduced, the viewability for the user isvery different, depending upon the arrangement. Thus, it is necessaryfor the image generation unit 30 to generate a two dimensional image (ora frame) in which the plurality of main photographic subjects (i.e. thetwo photographic subjects) are arranged in an appropriate manner.

The image processing unit 30 of this embodiment creates a twodimensional image in which the first photographic subject 301 facestoward the second photographic subject 302 in the image. In the imageshown by way of example in FIG. 11, the first photographic subject 301is facing in the rightward direction in the drawing. Thus, by creating atwo dimensional image (i.e. a frame) in which the first photographicsubject 301 and the second photographic subject 302 are arranged so thatthe first photographic subject 301 is facing toward the secondphotographic subject 302, it is possible to create an image (a picture)in which no sense of discomfort is imparted to the user. Accordingly theimage processing unit 30 creates a two dimensional image in which thefirst photographic subject 301 is arranged on the left side of thesecond photographic subject 302.

Next, the feature that the first photographic subject 301 is facingtoward the second photographic subject 302 in the image will beexplained. As described above, when a vector that describes theorientation of the main photographic subject in three dimensional spaceis projected upon the image (or upon the imaging surface), the projectedvector is the orientation of the main photographic subject in the image.For example, the vector shown in FIG. 11 is the projected vector for thefirst photographic subject 301. In the image 300, the first photographicsubject 301 is taken as being the origin, and the direction from thefirst photographic subject 301 toward the second photographic subject302 is taken as being the X axis. At this time, if the component of theprojected vector of the first photographic subject 301 in the X axisdirection is positive, then it can be determined that the firstphotographic subject 301 is facing the second photographic subject 302.Conversely, if the component of the projected vector of the firstphotographic subject 301 in the X axis direction is negative, then itcan be determined that the first photographic subject 301 is not facingthe second photographic subject 302.

The image generation unit 30 generates from the omnidirectional image atwo dimensional image that includes the first photographic subject 301and the second photographic subject 302 and in which the firstphotographic subject 301 is arranged on the first direction side of thesecond photographic subject 302, if the first distance by which theimage displayed upon the display unit 40 is shifted from when the firstphotographic subject 301 is displayed upon the display unit 40 until thesecond photographic subject 302 is displayed upon the display unit 40 islonger than the second distance by which the image displayed upon thedisplay unit 40 is shifted from when the second photographic subject 302is displayed upon the display unit 40 until the first photographicsubject 301 is displayed upon the display unit 40, and if, in the imagedisplayed upon the display unit 40 from when the second photographicsubject 302 is displayed upon the display unit 40 until the firstphotographic subject 301 is displayed upon the display unit 40, thefirst photographic subject 301 is not facing toward the secondphotographic subject 302.

Next, the method for determining the orientation of the photographicsubject in three dimensional space will be explained. While theorientation of the nose of a person has been explained as an example,the orientation of the face of the photographic subject (i.e. theperson) could also be employed. It would be acceptable for the directionin which the eyes are facing to be taken as being the orientation of theface, or, if the face is modeled as a plane, it may be taken as beingthe direction normal to that plane. Alternatively, not the orientationof the face of the person, but rather the orientation of the body of theperson may be employed as the orientation of the photographic subject.If the orientation of the body is employed, then the chest may bemodeled as a plane, and the direction normal to that plane may be takenas being the orientation of the body. In any of these cases, bystipulating the orientation of the photographic subject in threedimensional space in advance, it is possible uniquely to determine theorientation of the photographic subject in three dimensional space.Moreover, if a photographic subject other than a person is employed asthe main photographic subject, then an orientation that is appropriatefor that photographic subject may be stipulated. For example, if themain photographic subject is a vehicle or some other moving object, thenthe direction of traveling (i.e. the direction of movement) of thatvehicle may be taken as being the orientation of the main photographicsubject. Moreover, if the main photographic subject is a building, thenthe orientation of the main entrance at the front of that building maybe taken as being the orientation of the main photographic subject.

Next, the way in which the orientation of the photographic subject inthree dimensional space is acquired will be explained. Since, asdescribed above, the orientation is stipulated in dependence upon thephotographic subject, accordingly, for example, the image generationunit 30 is able to acquire the orientation of the main photographicsubject in three dimensional space by image analysis of theomnidirectional image, or from the output of a sensor providedseparately from the imaging unit 20, or by distance measurementcalculation employing the imaging unit 20. If it is possible to acquirea vector that indicates the orientation of the main photographic subjectin three dimensional space, then the projected vector can be acquired byprojecting that vector. And the image generation unit 30 is able tocalculate the orientation of the main photographic subject in the imageon the basis of the projected vector, as described above.

According to the embodiment described above, the following advantageousoperational effect is obtained.

(1) According to the structure of this embodiment, it is possibleautomatically to generate a two dimensional image that is suitable forviewing from the omnidirectional image.

Variant of Second Embodiment

It should be understood that a single device may incorporate two or moreof the imaging unit 20, the image generation unit 30, and the displayunit 40. For example, the imaging device 2 may incorporate the imagegeneration unit 30 in addition to the imaging unit 20. In this case, theimaging device 2 would also fulfil the role of the image processingdevice 3. Accordingly, the image processing device 3 may not be includedin the image processing system 1. And, as another example, the imageprocessing device 3 may incorporate the display unit 40 in addition tothe image generation unit 30. In this case, the image processing device3 would also fulfil the role of the reproduction device 4. Accordingly,the reproduction device 4 may not be included in the image processingsystem 1. As another example, in addition to the imaging unit 20, theimaging device 2 may also include the image generation unit 30 and thedisplay unit 40. In this case, the imaging device 2 would also combinethe roles of the image processing device 3 and the reproduction device4. In other words, the imaging device 2 would, by itself, providefunctions equivalent to those of the image processing system 1.

FIG. 10 is a block diagram schematically showing an electronic device1000 that serves as both the image processing device 3 and thereproduction device 4. This electronic device 1000 may be, for example,a smart phone or a tablet terminal. The electronic device 1000 comprisesan image generation unit 30, an input unit 31, an output unit 32, adisplay unit 40, a control unit 42, and an actuation unit 43. Theelectronic device 1000 is capable of creating a two dimensional video,reproducing upon the display unit 40 a two dimensional video that hasbeen created, storing upon the storage medium 52 a two dimensional videothat has been created, and performing reproduction of an omnidirectionalimage upon the display unit 40. It should be understood that theoperation of each of these sections of the electronic device 1000 is thesame as in the case of the first embodiment, and accordingly explanationthereof will be omitted.

According to the variant embodiment described above, the followingadvantageous operational effect is obtained.

(2) According to the structure of this embodiment, it is possible toobtain advantageous operational effects similar to those of theembodiment described above.

It should be understood that the creation of a two dimensional video bythe image generation unit 30 may be performed in real time in parallelwith the creation of the omnidirectional video by the imaging unit 20,or may be started after the creation of the omnidirectional video hasbeen completed. In a similar manner, the display of the two dimensionalvideo upon the display unit 40 may be performed in real time in parallelwith the creation of the two dimensional video by the image generationunit 30, or may be started after the creation of the two dimensionalvideo has been completed.

According to the variant embodiment described above, the followingadvantageous operational effect is obtained.

(3) According to the structure of this embodiment, it is possibleautomatically to generate a two dimensional image that is suitable forviewing from the all-around image.

Third Embodiment

In an image processing system according to the third embodiment, thedetails of the two dimensional video creation processing performed bythe image generation unit 30 are different from the case of the firstembodiment. It should be understood that features that are not mentionedin connection with this embodiment are the same as the details explainedin connection with the first embodiment. In other words, the detailsexplained in connection with the first embodiment are all to beconsidered as being incorporated in this third embodiment. In thefollowing, the image processing system according to the third embodimentwill be explained with emphasis centering upon the features ofdifference from the image processing system of the first embodiment. Theimage generation unit 30 of this third embodiment performs thephotographic subject specification processing in a similar manner to thecase in the first embodiment. Since the details thereof are the same asin the first embodiment, accordingly explanation will be omitted.

FIG. 12 and FIG. 13 are figures for explanation of the processing forcreation of a two dimensional image. FIG. 12(a) and FIG. 13(a) are planviews of a volleyball court. In the example of FIG. 12 and FIG. 13, theimaging device 2 is installed in the center of a court 400. Thesituation in the court 400 at the time point t1 is shown in FIG. 12(a),and a first frame photographed at the time point t1 (hereinafterreferred to as a “first omnidirectional image 500”) is shown in FIG.12(b). It will be supposed that, at this time, the image generation unit30 recognizes a photographic subject 403 (which hereinafter will bereferred to as a “third photographic subject 403”) that is a person asbeing the main photographic subject. Accordingly, according to the stepS35 in the flow chart of FIG. 8, the image generation unit 30 creates atwo dimensional image (i.e. a frame) that includes this thirdphotographic subject 403.

And the situation in the court 400 at the time point t2 subsequent tothe time point t1 is shown in FIG. 13(a), and a 31st frame photographedat the time point t2 (hereinafter referred to as a “secondomnidirectional image 510”) is shown in FIG. 13(b). At this time, thethird photographic subject 403 which was the main photographic subjectat the time point t1 as described above, and a main photographic subject404 which is a person (and which hereinafter will be referred to as a“fourth photographic subject 404”) and a main photographic subject 405which is a person (and which hereinafter will be referred to as a “fifthphotographic subject 405”) at the time point t2 are present in the court400. Thus, it will be supposed that the image generation unit 30specifies the fourth photographic subject 404 and the fifth photographicsubject 405 as being the two main photographic subjects. It will besupposed that, at the time point t2, the third photographic subject 403is not specified as being a main photographic subject.

The positional relationship between the imaging device 2, the fourthphotographic subject 404, and the fifth photographic subject 405 is thesame as in the example shown in FIG. 6. In other words, in FIG. 13(b),due to control to shift the part of the second omnidirectional image 510displayed upon the display unit 40 in the leftward direction, thedistance by which the image displayed upon the display unit 40 shiftsfrom when the fourth photographic subject 404 is displayed upon thedisplay unit 40 until the fifth photographic subject 405 is displayedupon the display unit 40 (i.e. the distance in the rightward directionfrom the fourth photographic subject 404 to the fifth photographicsubject 405 (hereinafter this will be termed the “first distance”)) islonger than the distance in the rightward direction from the fifthphotographic subject 405 until the fourth photographic subject 404(hereinafter this will be termed the “second distance”). Accordingly, ifit is supposed that a two dimensional image is created from the secondomnidirectional image 510 by performing processing similar to thatperformed in the first embodiment, then a two dimensional image will becreated in which the fifth photographic subject 405 is positioned on theleft side of the fourth photographic subject 404.

By contrast, the image generation unit 30 of this embodiment generates atwo dimensional image while considering the position of the mainphotographic subject in the previous frame in time, in other words inthe first omnidirectional image 500 that was captured at the time pointt1 shown in FIG. 12(b). In concrete terms, the image generation unit 30specifies an angle of view 401 that is the angle of view including themain photographic subject (i.e. the third photographic subject 403) atthe time point t1. And the image generation unit 30 specifies a partialimage 511 at the position corresponding to the angle of view 401 fromthe second omnidirectional image 510 that was captured at the time pointt2. Then the image generation unit 30 creates a two dimensional image inwhich the fourth photographic subject 404 and the fifth photographicsubject 405 are arranged, on the basis of the positional relationshipbetween the partial image 511, the fourth photographic subject 404, andthe fifth photographic subject 405 in the second omnidirectional image510.

The processing performed by the image generation unit 30 to create a twodimensional image will now be described in detail. The image generationunit 30 defines hypothetically, in the second omnidirectional image, apartial image 511 at a position corresponding to the partial imageincluding the third photographic subject 403 in the firstomnidirectional image (in other words, a partial image in the secondomnidirectional image corresponding to the angle of view 401,hereinafter termed a “first partial image 511”). And the imagegeneration unit 30 defines hypothetically a partial image in which thefourth photographic subject 404, the fifth photographic subject 405, andthe first partial image 511 between the fourth photographic subject 404and the fifth photographic subject 405 are included (in other words, apartial image in the second omnidirectional image corresponding to anangle of view 406, hereinafter termed a “second partial image”). Thenthe image processing unit 30 creates a two dimensional image such thatthe left and right positional relationship of the fourth photographicsubject 404 and the fifth photographic subject 405 in the second partialimage is maintained.

Now, it should be understood that the reason why the two dimensionalimage is generated while considering the position of the mainphotographic subject in the first omnidirectional image 500 that wascaptured at the time point t1 is as follows. The image generation unit30 generates a two dimensional image (i.e. a frame) including the thirdphotographic subject 403 from the first omnidirectional image 500 thatwas captured at the time point t1. That is, over a predetermined timeperiod, the user views the direction corresponding to the angle of view401 as the video image reproduced from the two dimensional video. Then,at the time point t2, the main photographic subject changes from thethird photographic subject 403 to the fourth photographic subject 404and the fifth photographic subject 405. As a result, the imagegeneration unit 30 generates a two dimensional image (i.e. a frame)including the fourth photographic subject 404 and the fifth photographicsubject 405 from the second omnidirectional image 510 captured at thetime point t2. This is because, from the point of view of a user who hasviewed the direction corresponding to the angle of view 401 for thepredetermined period as the video image reproduced from the twodimensional video, it is easier to understand the arrangement of thephotographic subjects in three dimensional space by arranging the fourthphotographic subject 404 and the fifth photographic subject 405 whiletaking the position of the third photographic subject 403 (in otherwords, the direction corresponding to the angle of view 401) asreference. In other words, in FIG. 13(a), when the direction of theangle of view 401 as viewed from the imaging device 2 is taken asreference, the fourth photographic subject 404 is present on the leftand the fifth photographic subject 405 is present on the right.Accordingly, during reproduction of the two dimensional video, whenchanging over from the two dimensional image (i.e. the frame) thatincludes the third photographic subject 403 to the two dimensional image(i.e. the frame) that includes the fourth photographic subject 404 andthe fifth photographic subject 405, by arranging the fourth photographicsubject 404 on the left and the fifth photographic subject on the rightin the two dimensional image, it is possible to reduce any sense ofdiscomfort imparted to the user.

By determining the angle of view in consideration of the angle of viewof the previous frame in time in this manner, it is possible to avoidconfusion during viewing due to a sudden change of scene. In otherwords, if reproduction is suddenly performed in a direction that iscompletely different from that of the previous frame, then there is apossibility that it may become difficult to know from which portion ofthe omnidirectional image the screen has been cut out. However since, asdescribed above, the image generation unit 30 of this embodimentdetermines the angle of view in consideration of the angle of view ofthe previous frame, accordingly it becomes possible to create a twodimensional video in which the spatial relationships can be easilyapprehended, and that accordingly is suitable for viewing

It should be understood that “the left and right positional relationshipis maintained” means that only the positional relationship in the leftand right direction is considered and its relationship is maintainedwhile ignoring the up and down positional relationship. In other wordsthe left and right positional relationship is considered to bemaintained as long as the left and right positional relationship ismaintained, regardless of how much the up and down positionalrelationship changes.

According to the embodiment described above, the following advantageousoperational effect is obtained.

(1) According to the structure of this embodiment, it is possibleautomatically to generate a two dimensional image that is suitable forviewing from the omnidirectional image.

Variant of Third Embodiment

It should be understood that a single device may incorporate two or moreof the imaging unit 20, the image generation unit 30, and the displayunit 40. For example, the imaging device 2 may incorporate the imagegeneration unit 30 in addition to the imaging unit 20. In this case, theimaging device 2 would also fulfil the role of the image processingdevice 3. Accordingly, the image processing device 3 may not be includedin the image processing system 1. And, as another example, the imageprocessing device 3 may incorporate the display unit 40 in addition tothe image generation unit 30. In this case, the image processing device3 would also fulfil the role of the reproduction device 4. Accordingly,the reproduction device 4 may not be included in the image processingsystem 1. As another example, in addition to the imaging unit 20, theimaging device 2 may also include the image generation unit 30 and thedisplay unit 40. In this case, the imaging device 2 would also fulfilthe roles of the image processing device 3 and the reproduction device4. In other words, the imaging device 2 would, by itself, providefunctions equivalent to those of the image processing system 1.

FIG. 10 is a block diagram schematically showing an electronic device1000 that combines the image processing device 3 and the reproductiondevice 4. This electronic device 1000 may be, for example, a smart phoneor a tablet terminal. The electronic device 1000 comprises an imagegeneration unit 30, an input unit 31, an output unit 32, a display unit40, a control unit 42, and an actuation unit 43. The electronic device1000 is capable of creating a two dimensional video, reproducing uponthe display unit 40 a two dimensional video that has been created,storing upon the storage medium 52 a two dimensional video that has beencreated, and performing reproduction of an omnidirectional video uponthe display unit 40. It should be understood that the operation of eachof these sections of the electronic device 1000 is the same as in thecase of the first embodiment, and accordingly explanation thereof willbe omitted.

According to the variant embodiment described above, the followingadvantageous operational effect is obtained.

(2) According to the structure of this embodiment, it is possible toobtain advantageous operational effects similar to those of theembodiments described above.

It should be understood that the creation of a two dimensional video bythe image generation unit 30 may be performed in real time in parallelwith the creation of the omnidirectional video by the imaging unit 20,or may be started after the creation of the omnidirectional video hasbeen completed. In a similar manner, the display of the two dimensionalvideo upon the display unit 40 may be performed in real time in parallelwith the creation of the two dimensional video by the image generationunit 30, or may be started after the creation of the two dimensionalvideo has been completed.

According to the variant embodiment described above, the followingadvantageous operational effect is obtained.

(3) According to the structure of this embodiment, it is possibleautomatically to generate a two dimensional image that is suitable forviewing from the all-around image.

Although an example has been explained in which two photographicsubjects arranged along the left and right direction are present, thesame would be the case if two photographic subjects are present and arearranged along some direction other than the left and right direction.Furthermore, if the main photographic subject shifts, not only in theleft and right direction but also in the up and down direction, it wouldbe acceptable to create a two dimensional image in which the up and downpositional relationship is maintained, instead of the left and rightpositional relationship being maintained.

The disclosure of the following application, from which priority isclaimed, is incorporated herein by reference:

Japanese Patent Application No. 2017-48861 (filed on Mar. 14, 2017).

REFERENCE SIGNS LIST

1: image processing system, 2: imaging device, 3: image processingdevice, 4: reproduction device, 20: imaging unit, 30: image generationunit, 31: input unit, 40: display unit, 42: control unit.

1. (canceled)
 2. (canceled)
 3. An image processing device, comprising:an input unit through which are inputted first image data which is aportion of an image in which a first photographic subject and a secondphotographic subject are imaged, and which is employed for the secondphotographic subject to be displayed after the first photographicsubject has been displayed, and for the first photographic subject thenagain to be displayed upon the display unit by repeating control toshift a portion of the image displayed upon the display unit in a firstdirection and to display a portion of the image that is not displayedupon the display unit; and an image generation unit that generates, fromthe first image data, second image data including the first photographicsubject and the second photographic subject, based on a first distanceby which the image displayed upon the display unit shifts from the firstphotographic subject being displayed upon the display unit until thesecond photographic subject is displayed upon the display unit, and asecond distance by which the image displayed upon the display unitshifts from the second photographic subject being displayed upon thedisplay unit until the first photographic subject is displayed upon thedisplay unit.
 4. The image processing device according to claim 3,wherein: if the first distance is longer than the second distance, theimage generation unit generates, from the first image data, the secondimage data that includes the first photographic subject and the secondphotographic subject, and in which the second photographic subject isarranged towards the first direction from the first photographicsubject.
 5. (canceled)
 6. An electronic device, comprising: a displayunit that displays first image data in which a first photographicsubject and a second photographic subject are imaged; a control unitthat displays the second photographic subject after the firstphotographic subject has been displayed, and then displays the firstphotographic subject again upon the display unit, by repeating controlto shift a portion of the first image data displayed upon the displayunit in a first direction and to displays a portion of the first imagedata that is not displayed upon the display unit; and an imagegeneration unit that generates, from the first image data, second imagedata in which the first photographic subject and the second photographicsubject are arranged based on a first distance by which the imagedisplayed upon the display unit shifts from the first photographicsubject being displayed upon the display unit until the secondphotographic subject is displayed upon the display unit, and a seconddistance by which the image displayed upon the display unit shifts fromthe second photographic subject being displayed upon the display unituntil the first photographic subject is displayed upon the display unit.7. The electronic device according to claim 6, wherein: if the firstdistance is longer than the second distance, the image generation unitgenerates, from the first image data, the second image data thatincludes the first photographic subject and the second photographicsubject, and in which the second photographic subject is arrangedtowards the first direction from the first photographic subject. 8.(canceled)
 9. An image processing device, comprising: an input unitthrough which is inputted an all-around image including a firstphotographic subject and a second photographic subject that have beenimaged by an imaging unit; and an image generation unit that generatesfrom the all-around image an image in which the first photographicsubject and the second photographic subject are arranged based on ashortest path in the all-around image from the first photographicsubject to the second photographic subject.
 10. The image processingdevice according to claim 9, wherein: the image generation unitspecifies a direction from the first photographic subject towards thesecond photographic subject as being a first direction in a partialimage of the all-around image in which the first photographic subject,the second photographic subject, and a third photographic subject thatis present in the shortest path are included, and generates from the allaround-image the image that includes the first photographic subject andthe second photographic subject, and in which the second photographicsubject is arranged towards the first direction from the firstphotographic subject.
 11. The image processing device according to claim3, wherein: the control to display a portion of the image that is notdisplayed upon the display unit is control to display at least a part ofthe portion of the image that is not displayed upon the display unit.12. The image processing device according to claim 6, wherein: thecontrol to display a portion of the image that is not displayed upon thedisplay unit is control to display at least a part of the portion of theimage that is not displayed upon the display unit.